Method and system for alignment and insertion of wire contact with wire contact insertion holes of a connector

ABSTRACT

A method, system and computer program product are provided for aligning and inserting a wire contact within a target hole of a connector to facilitate the automated insertion of the wire ends of a wire bundle assembly into the wire contact insertion holes of a connector. Methods may include: obtaining captured images, from at least two image capture devices attached to an end-effector of a robot, of a wire gripper of the end-effector; causing the robot to advance the end-effector to move the wire contact within a predetermined distance of the connector; causing the robot to advance the end-effector to move the wire contact toward the connector a predetermined additional amount more; and identifying, based on movement of the wire contact the predetermined additional amount more, if alignment is correct from force feedback at the wire gripper.

TECHNICAL FIELD

A method, system and computer program product are provided in accordancewith an example embodiment in order to align wire contacts withinsertion holes of a connector, and more particularly, to using amachine-vision system to automatically align wire contacts with wirecontact insertion holes of a connector and to insert the wire contactsinto respective holes of the connector.

BACKGROUND

Wire bundles consisting of a plurality of wires are utilized in avariety of industries to carry a myriad of different types of signals.The wire of a wire bundle assembly must frequently be terminated with awire contact and the resulting wire end is inserted into a wire contactinsertion hole of a connector, such as in a rubber grommet of aconnector. As each wire of a wire bundle is unique and may carry adifferent type of signal, the wire ends of a wire bundle assembly mustbe inserted into specific wire contact insertion holes of a connector inorder to make the proper connections.

The wire ends of a wire bundle assembly may be manually inserted intothe respective wire contact insertion holes defined by a connector. Aswire bundle assemblies commonly include hundreds of wires, this manualconnection process may be relatively time consuming and error prone and,as a result, may increase the cost of the overall assembly including thewire bundle assembly. As such, automated techniques to insert the wireends of a wire bundle assembly into the wire contact insertion holes ofa connector have been developed in an effort to reduce the time expendedto make the connections and to correspondingly reduce the cost of theresulting assembly. However, wire bundle assembly machines generallyrequire the connectors to be in a very restricted and controlled set oflocations in order to increase the likelihood that the wire ends of thewire bundle assembly may be properly inserted into the wire contactinsertion holes of the connector. As such, wire bundle assembly machineslimit the flexibility with which connectors may be presented and, assuch, are not suitable for all scenarios.

BRIEF SUMMARY

A method, system and computer program product are provided for aligningwire contacts with insertion holes defined by a connector so as tofacilitate the automated insertion of the wire ends of a wire bundleassembly into the wire contact insertion holes of a connector. Byfacilitating the automated insertion of the wire ends of a wire bundleassembly into the wire contact insertion holes of a connector, the timerequired to establish such connections and correspondingly the costassociated with the resulting assembly may be reduced while reducing theerror rate associated with the connections. The method, system andcomputer program product of an example embodiment provide substantialflexibility in relation to the manner in which the connector may belocated, while still permitting the wire ends of a wire bundle assemblyto be securely inserted into and electrically connected to theappropriate wire contact insertion holes of the connector.

In an example embodiment, a system is provided to align wire contactswith insertion holes of a connector and to insert the wire connectorsinto respective holes of the connector. The system may include a robothaving an end-effector, where the end-effector includes a wire gripperand at least two image capture devices secured to the end-effector; anda computing device. The computing device may be configured to: processimages captured by the image capture devices to establish a correctivetransformation to align the wire contact with the target hole of theconnector in a connector surface; cause the robot to translate theend-effector to move the wire gripper along the correctivetransformation; cause the robot to advance the end-effector untilcontact is made with the connector surface; identify, based on forcemeasurements, if the wire contact is inside the target hole of theconnector; cause the robot to advance the end-effector to move the wirecontact toward the connector a predetermined additional amount more; andidentify, based on movement of the wire contact the predeterminedadditional amount more, if alignment is correct from force feedback atthe wire gripper.

According to some embodiments, the computing device configured toidentify, based on force measurements, if the wire contact is inside thetarget hole of the connector, is configured to: identify that the wirecontact is not aligned with the target hole of the connector in responseto a force observed at the wire gripper above a predefined value; andidentify that the wire contact is aligned with the target hole of theconnector in response to the force observed at the wire gripper beingbelow the predefined value. The computing device configured to processimages captured by the image capture devices to establish correctivetransformation to align the wire contact with the target hole of theconnector may be configured to: identify, within the processed images, atip of the wire contact and a direction along which the wire contactextends; and establish a corrective transformation to align the tip ofthe wire contact and the direction along which the wire contact extendswith the target hole in an axis of a three-dimensional coordinate systemof the robot end-effector from the processed images.

The computing device of some embodiments may be configured to: determinea maximum distance the wire gripper can advance toward the connector;cause the robot to advance the end-effector to move the wire grippertoward the connector; as the robot is caused to advance the end-effectorto move the wire gripper toward the connector, in response to forcefeedback on the wire gripper satisfying an insertion force value,determine that the wire contact is fully inserted into the connector;and as the robot is caused to advance the end-effector to move the wiregripper toward the connector, in response to force feedback on the wiregripper failing to satisfy the insertion force value before reaching themaximum distance the wire gripper can advance toward the connector,determine that the wire contact is not fully inserted into theconnector. In response to the computing device determining that the wirecontact is fully inserted into the connector, the computing device isfurther configured to: cause the robot to retract the end-effector tomove the wire gripper away from the connector; and in response to thewire gripper moving a pull distance without force feedback on the wiregripper exceeding a pull test value, identify the wire contact asimproperly inserted; in response to force feedback on the wire gripperexceeding a pull test value without the wire gripper having moved a pulldistance, identify the wire contact as properly inserted.

According to some embodiments, in response to the computing devicedetermining that the contact is not fully inserted into the connector,the computing device may further be configured to: cause the wiregripper to release a grip on the wire contact; cause the robot toretract the end-effector to move the wire gripper away from theconnector; cause the wire gripper to re-grip the wire contact; and causethe robot to advance the end-effector to move the wire gripper and thewire contact toward the connector. According to some exampleembodiments, force feedback may be established based on a force sensordisposed between the robot and the wire gripper. A tool interfaces withforce torque sensor may be employed between the robot and the endeffector in such a way that the force torque sensor may be able toaccurately detect forces in three orthogonal axes and torques about eachof those three axes. Another example embodiment which may be usedinstead of or in combination with a force torque sensor arrangement is astrain gauge in the gripper, or a strain gauge array. A single straingauge may not be able to establish the axis along which a force isreceived but may be used in an example embodiment to detect the forcefeedback of a wire contact inserted or pulled from a connector. A straingauge array may be used to identify the axes along which forces arereceived if desired. The computing device of some embodiments may beconfigured to: identify, in images captured by the image capturedevices, the orientation of a visible portion of the wire contact; andcompute a corrective movement in a coordinate frame of the robot endeffector using the orientation of the wire contact.

Embodiments provided herein may include a method to align and insert awire contact into a target hole of a connector. Methods may include:obtaining captured images, from at least two image capture devicesattached to an end-effector of a robot, of a wire gripper of theend-effector; processing images captured by the image capture devices toestablish a corrective transformation to align the wire contact with thetarget hole of the connector in a connector surface; causing the robotto translate the end-effector to move the wire gripper along thecorrective transformation; causing the robot to advance the end-effectoruntil contact is made with the connector surface; identify, based onforce measurements, if the wire contact is inside the target hole of theconnector; causing the robot to advance the end-effector to move thewire contact toward the connector a predetermined additional amountmore; and identifying, based on movement of the wire contact thepredetermined additional amount more, if alignment is correct from forcefeedback at the wire gripper.

According to some embodiments, identifying, based on force measurements,if the wire contact is inside the target hole of the connector mayinclude: identifying that the wire contact is not aligned with thetarget hole of the connector in response to a force observed at the wiregripper above a predefined value; and identifying that the wire contactis aligned with the target hole of the connector in response to theforce observed at the wire gripper being below the predefined value.Processing images captured by the image capture devices to establishcorrective transformation to align the wire contact with the target holeof the connector may include: identifying, within the processed images,a tip of the wire contact and a direction along which the wire contactextends; and establishing a corrective transformation to align the tipof the wire contact and the direction along which the wire contactextends with the target hole in an axis of a three-dimensionalcoordinate system of the robot end-effector from the processed images.

Methods may include: determining a maximum distance the wire gripper canadvance toward the connector; causing the robot to advance theend-effector to move the wire gripper toward the connector; as the robotis caused to advance the end-effector to move the wire gripper towardthe connector, in response to force feedback on the wire grippersatisfying an insertion force value, determining that the wire contactis fully inserted into the connector; and as the robot is caused toadvance the end-effector to move the wire gripper toward the connector,in response to force feedback on the wire gripper failing to satisfy theinsertion force value before reaching the maximum distance the wiregripper can advance toward the connector, determining that the wirecontact is not fully inserted into the connector.

According to some embodiments, in response to determining that the wirecontact is fully inserted into the connector, methods may include:causing the robot to retract the end-effector to move the wire gripperaway from the connector; and in response to the wire gripper moving apull distance without force feedback on the wire gripper exceeding apull test value, identifying the wire contact as improperly inserted; inresponse to force feedback on the wire gripper exceeding a pull testvalue without the wire gripper having moved a pull distance, identifyingthe wire contact as properly inserted. In response to determining thatthe wire contact is not fully inserted into the connector, methods mayinclude: causing the wire gripper to release a grip on the wire contact;causing the robot to retract the end-effector to move the wire gripperaway from the connector; causing the wire gripper to re-grip the wirecontact; and causing the robot to advance the end-effector to move thewire gripper and the wire contact toward the connector. Force feedbackat the wire gripper may be established based on a force sensor betweenthe robot and the wire gripper. Methods may optionally include:identifying, in images captured by the image capture devices, theorientation of a visible portion of the wire contact; and computing acorrective movement in a coordinate frame of the robot end effectorusing the orientation of the wire contact.

Embodiments provided herein may include a computer program product foraligning and inserting a wire contact into a target hole defined by aconnector. The computer program product may include at least onenon-transitory computer-readable storage medium havingcomputer-executable program code instructions stored therein. Thecomputer-executable program code instructions including program codeinstructions to: obtain captured images from at least two image capturedevices attached to an end-effector of a robot, of a wire gripper of theend-effector; process images captured by the image capture devices toestablish a corrective transformation to align the wire contact with thetarget hole of the connector in a connector surface; cause the robot totranslate the end-effector to move the wire gripper along the correctivetransformation; cause the robot to advance the end-effector untilcontact is made with the connector surface; identify, based on forcemeasurements, if the wire contact is inside the target hole of theconnector; cause the robot to advance the end-effector to move the wirecontact toward the connector a predetermined additional amount more; andidentify, based on movement of the wire contact the predeterminedadditional amount, if alignment is correct from the force feedback atthe wire gripper.

The program code instructions to identify, based on force measurements,if the wire contact is inside the target hole of the connector, mayinclude program code instructions to: identify that the wire contact isnot aligned with the target hole of the connector in response to a forceobserved at the wire gripper above a predefined value; and identify thatthe wire contact is aligned with the target hole of the connector inresponse to the force observed at the wire gripper below the predefinedvalue. The program code instructions to process images captured by theimage capture devices to establish corrective transformation to alignthe wire contact with the target hole of the connector may includeprogram code instructions to: identify, within the processed images, atip of the wire contact and a direction along which the wire contactextends; and establish a corrective transformation to align the tip ofthe wire contact and the direction along which the wire contact extendswith the target hole in an axis of a three-dimensional coordinate systemof the robot end-effector from the processed images.

According to some embodiments, the computer program product may includeprogram code instructions to: determine a maximum distance the wiregripper can advance toward the connector; cause the robot to advance theend-effector to move the wire gripper toward the connector; as the robotis caused to advance the end-effector to move the wire gripper towardthe connector, in response to force feedback on the wire grippersatisfying an insertion force value, determine that the wire contact isfully inserted into the connector; and as the robot is caused to advancethe end-effector to move the wire gripper toward the connector, inresponse to force feedback on the wire gripper failing to satisfy theinsertion force value before reaching the maximum distance the wiregripper can advance toward the connector, determine that the wirecontact is not fully inserted into the connector.

In response to determining that the wire contact is fully inserted intothe connector, according to some embodiments, the computer programproduct may include program code instructions to: cause the robot toretract the end-effector to move the wire gripper away from theconnector; and in response to the wire gripper moving a pull distancewithout force feedback on the wire gripper exceeding a pull test value,identify the wire contact as properly inserted; in response to forcefeedback on the wire gripper exceeding a pull test value without thewire gripper having moved a pull distance, identify the wire contact asproperly inserted. In response to determining that the wire contact isnot fully inserted into the connector, the computer program product mayinclude program code instructions to: cause the wire gripper to releasea grip on the wire contact; cause the robot to retract the end-effectorto move the wire gripper away from the connector; cause the wire gripperto re-grip the wire contact; and cause the robot to advance theend-effector to move the wire gripper and the wire contact toward theconnector.

Embodiments provided herein may include an apparatus to align and inserta wire contact into a target hole of a connector. An example apparatusmay include: means for obtaining captured images, from at least twoimage capture devices attached to an end-effector of a robot, of a wiregripper of the end-effector; means for processing images captured by theimage capture devices to establish a corrective transformation to alignthe wire contact with the target hole of the connector in a connectorsurface; means for causing the robot to translate the end-effector tomove the wire gripper along the corrective transformation; means forcausing the robot to advance the end-effector until contact is made withthe connector surface; identify, based on force measurements, if thewire contact is inside the target hole of the connector; means forcausing the robot to advance the end-effector to move the wire contacttoward the connector a predetermined additional amount more; and meansfor identifying, based on movement of the wire contact the predeterminedadditional amount more, if alignment is correct from force feedback atthe wire gripper.

According to some embodiments, the means for identifying, based on forcemeasurements, if the wire contact is inside the target hole of theconnector, may include: means for identifying that the wire contact isnot aligned with the target hole of the connector in response to a forceobserved at the wire gripper above a predefined value; and means foridentifying that the wire contact is aligned with the target hole of theconnector in response to the force observed at the wire gripper beingbelow the predefined value. The means for processing images captured bythe image capture devices to establish corrective transformation toalign the wire contact with the target hole of the connector mayinclude: means for identifying, within the processed images, a tip ofthe wire contact and a direction along which the wire contact extends;and means for establishing a corrective transformation to align the tipof the wire contact and the direction along which the wire contactextends with the target hole in an axis of a three-dimensionalcoordinate system of the robot end-effector from the processed images.

Embodiments of an apparatus may include: means for determining a maximumdistance the wire gripper can advance toward the connector; means forcausing the robot to advance the end-effector to move the wire grippertoward the connector; as the robot is caused to advance the end-effectorto move the wire gripper toward the connector, in response to forcefeedback on the wire gripper satisfying an insertion force value, meansfor determining that the wire contact is fully inserted into theconnector; and as the robot is caused to advance the end-effector tomove the wire gripper toward the connector, in response to forcefeedback on the wire gripper failing to satisfy the insertion forcevalue before reaching the maximum distance the wire gripper can advancetoward the connector, means for determining that the wire contact is notfully inserted into the connector.

According to some embodiments, in response to determining that the wirecontact is fully inserted into the connector, an apparatus may include:means for causing the robot to retract the end-effector to move the wiregripper away from the connector; and in response to the wire grippermoving a pull distance without force feedback on the wire gripperexceeding a pull test value, means for identifying the wire contact asimproperly inserted; in response to force feedback on the wire gripperexceeding a pull test value without the wire gripper having moved a pulldistance, means for identifying the wire contact as properly inserted.In response to determining that the wire contact is not fully insertedinto the connector, an apparatus may include: means for causing the wiregripper to release a grip on the wire contact; means for causing therobot to retract the end-effector to move the wire gripper away from theconnector; means for causing the wire gripper to re-grip the wirecontact; and means for causing the robot to advance the end-effector tomove the wire gripper and the wire contact toward the connector. Forcefeedback at the wire gripper may be established based on a force sensorbetween the robot and the wire gripper. An example apparatus may furtherinclude: means for identifying, in images captured by the image capturedevices, the orientation of a visible portion of the wire contact; andcomputing a corrective movement in a coordinate frame of the robot endeffector using the orientation of the wire contact.

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described certain example embodiments of the presentdisclosure in general terms, reference will hereinafter be made to theaccompanying drawings which are not necessarily drawn to scale, andwherein:

FIG. 1 is a perspective view of a connector according to an exampleembodiment of the present disclosure;

FIG. 2 is a front view of the connector of FIG. 1 according to anexample embodiment of the present disclosure;

FIG. 3 is a block diagram of the system that may be specificallyconfigured in accordance with an example embodiment of the presentdisclosure;

FIG. 4 depicts a robot end-effector, wire gripper, and image capturedevices according to an example embodiment of the present disclosure;

FIG. 5 illustrates images of a connector captured by the image capturedevices of the robot end-effector of FIG. 4 according to an exampleembodiment of the present disclosure;

FIG. 6 is a flowchart of a calibration routine for calibrating the imagecapture devices relative to the wire gripper and robot end-effectoraccording to an example embodiment of the present disclosure;

FIG. 7 is a flowchart of a process for aligning a wire contact with atarget insertion hole according to an example embodiment of the presentdisclosure;

FIG. 8 illustrates the process to extract the wire contact direction andtip position from an image according to an example embodiment of thepresent disclosure;

FIG. 9 illustrates the process flow for detecting contact holes inconnectors according to an example embodiment of the present disclosure;

FIG. 10 is a flowchart of a process for aligning the wire contactdirection with a target hole of the connector according to an exampleembodiment of the present disclosure;

FIG. 11 illustrates a connector with a wire bundle attached theretousing example embodiments of the alignment technique described herein;

FIG. 12 is a flowchart of a process for aligning a wire contact with atarget hole of a connector according to an example embodiment of thepresent disclosure;

FIG. 13 is a process flow of a method for aligning and inserting wirecontacts with target holes of a connector according to an exampleembodiment of the present disclosure; and

FIG. 14 is a flowchart of a process for aligning and inserting a wirecontact with a target hole of a connector according to an exampleembodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure now will be described more fully hereinafter withreference to the accompanying drawings, in which some, but not allaspects are shown. Indeed, the disclosure may be embodied in manydifferent forms and should not be construed as limited to the aspectsset forth herein. Rather, these aspects are provided so that thisdisclosure will satisfy applicable legal requirements. Like numbersrefer to like elements throughout.

A method, system, and computer program product are provided inaccordance with an example embodiment described herein to align wireends/contacts with corresponding insertion holes of a connector, such asin a rubber grommet of a connector, to facilitate automatic robotic wireinsertion. The process described herein detects wire contact andinsertion holes simultaneously using robotic-end-effector-mountedcameras. Using simultaneous detection, embodiments of the disclosedmethod provide feedback for corrective movements of a robot arm used toinsert the wire contacts into the insertion holes of the connector. Themovements of the robot arm align the wire contact with a targetinsertion hole for successful insertion into an appropriate hole of aconnector. The simultaneous detection described herein includes amulti-step process to detect the contact holes which may use colorfiltering, computing distances to a hole template, finding extrema in adistance map, and matching a hole mask to these extrema, as describedfurther below.

The assembly of wire bundles including the attachment of one or morewire connectors to the wire bundle has traditionally been alabor-intensive process that is both time consuming and introducesopportunities for errors in the assembly. Embodiments described hereinenable the automatic assembly of wire bundles and their associated wireconnectors. In particular, embodiments provide for the automaticinsertion of wire ends into connectors. Embodiments described herein mayuse a robot arm with an end-effector to insert the wires, supporting aflexible layout of connectors and wires.

A method, system and computer program product are provided in accordancewith an example embodiment in order to identify wire contacts and wirecontact insertion holes defined by a connector in order to align andinsert the wire contacts into the wire contact insertion holes. Althoughthe method, system and computer program product may be configured toidentify the wire contacts and wire contact insertion holes of a varietyof different types of connectors, the connectors generally define aplurality of wire contact insertion holes within a housing with the wirecontact insertion holes being arranged in a predefined configuration.Different connectors may include different numbers of wire contactinsertion holes and may include wire contact insertion holes arranged indifferent configurations.

One example of a connector is depicted in FIGS. 1 and 2 in the form of aconnector 10. As shown, the connector 10 includes a housing 12 and arubber grommet 16 disposed therein. Although the housing 12 may beconfigured differently for other types of connectors, the housing of theconnector 10 of the embodiment of FIGS. 1 and 2 is externally threadedto facilitate, for example, the secure threaded engagement of a wirebundle assembly or another connector therewith. The connector 10 ofFIGS. 1 and 2 also includes a radially extending flange defining aplurality of openings 14, such as for receiving screws or otherfasteners for mounting the connector to an assembly. Although theconnector 10 of FIG. 1 has a cylindrical shape, the connector of otherexample embodiments may have different sizes and shapes. In regards tothe example connector of FIGS. 1 and 2, a rubber grommet 16 is disposedwithin the housing and the rubber grommet defines a plurality of wirecontact insertion holes 18. The wire contact insertion holes 18 definedby the rubber grommet 16 are configured, e.g., sized and shaped, suchthat a wire end consisting of a wire contact connected, e.g., crimped,to the end of a wire, is inserted into and mechanically retained withinthe wire contact insertion hole 18. In some, but not all embodiments,the rubber grommet may also include a plurality of wire contacts inalignment with respective wire contact insertion holes defined by therubber grommet such that the wire end may be brought into secureelectrical contact with a respective wire contact of the connector.

As shown by the example of the connector 10 of FIGS. 1 and 2, theplurality of wire contact insertion holes 18 defined by the rubbergrommet 16, are arranged in a predefined pattern. In some embodiments,not all of the wire contact insertion holes of a connector 10 will beutilized and, instead, only a subset of the wire contact insertion holeswill receive and make electrical connection with corresponding wire endsof the wire bundle assembly. As illustrated in FIG. 2, the wire contactinsertion holes 18 defined by the rubber grommet 16 that are not to beutilized in conjunction with a particular application may be eliminatedfrom further consideration by the insertion a plug 20 into therespective wire contact insertion hole defined by the rubber grommet.Although a connector 10 that may be analyzed in accordance with anexample embodiment of the present disclosure is depicted in FIGS. 1 and2 and will be described hereinafter, the method, system and computerprogram product of an example embodiment may be utilized in conjunctionwith a wide variety of other connectors and the connector is illustratedand described by way of example, but not of limitation.

Referring now to FIG. 3, a system for identifying wire contact insertionholes of a connector 10 is depicted. As shown, the system 30 includescameras 32 configured the capture images of the connector 10. Whileplural cameras are indicated in FIG. 3, embodiments may employ a singlecamera, or may employ a single camera operating with mirrors to providevarious perspectives of the connector 10 using a single camera. Thecameras described herein are a type of image capture device, where avariety of image capture device types may be used in place of a camera.Image capture devices, generally, capture an image of the field of viewof the device. A camera, as described herein, captures an image of thefield of view in the visible light spectrum and processes the imageaccordingly. The cameras 32 may be configured to capture a gray scaleimage of the connector 10. Alternatively, the cameras 32 may beconfigured to capture color images of the connector 10. In an embodimentin which color images of the connector 10 are captured, the imageassociated with each different color channel of the cameras 32, such asthe red, green and blue color channels, may be averaged to create acomposite image for subsequent analysis and review. Alternatively, thedifferent color channels of the cameras 32 may be separately analyzed.The cameras 32 are generally configured to capture images of the frontface of the connector 10, such as shown in FIG. 2, such that theplurality of wire contact insertion holes 18 defined by the rubbergrommet 16 are clearly visible. The cameras 32 may also be configured tocapture images of the wire contacts during alignment of the wirecontacts with the connector 10. As such, the image captured by thecameras 32 of an example embodiment may be captured at a plurality ofangles to provide different perspectives of the connector 10 and wirecontacts.

In addition to the cameras 32, the system 30 of FIG. 3 includes acomputing device 34 configured to analyze the images of the connector 10captured by the cameras and to identify wire contact insertion holes ofthe connector and wire contacts. As also shown in FIG. 3, the system 30of an example embodiment also includes or is in communication with arobot 44 and, more particularly, a robotic end effector that is utilizedto insert wire ends/contacts into respective candidate contact insertionholes of the connector 10 based upon the identification of the wirecontact insertion holes of the connector and the wire contacts by thecomputing device 34.

The computing device 34 may be configured in various manners and, assuch, may be embodied as a personal computer, a tablet computer, acomputer workstation, a mobile computing device such as a smartphone, aserver or the like. Regardless of the manner in which the computingdevice 34 is embodied, the computing device of an example embodimentincludes or is otherwise associated with processing circuitry 36, memory38, and optionally a user interface 40 and a communication interface 42for performing the various functions herein described. The processingcircuitry 36 may, for example, be embodied as various means includingone or more microprocessors, one or more coprocessors, one or moremulti-core processors, one or more controllers, one or more computers,various other processing elements including integrated circuits such as,for example, an ASIC (application specific integrated circuit) or FPGA(field programmable gate array), or some combination thereof. In someexample embodiments, the processing circuitry 36 is configured toexecute instructions stored in the memory 38 or otherwise accessible tothe processing circuitry. These instructions, when executed by theprocessing circuitry 36, may cause the computing device 34 and, in turn,the system 30 to perform one or more of the functionalities describedherein. As such, the computing device 34 may comprise an entity capableof performing operations according to an example embodiment of thepresent disclosure while configured accordingly. Thus, for example, whenthe processing circuitry 36 is embodied as an ASIC, FPGA or the like,the processing circuitry and, correspondingly, the computing device 34may comprise specifically configured hardware for conducting one or moreoperations described herein. Alternatively, as another example, when theprocessing circuitry 36 is embodied as an executor of instructions, suchas may be stored in the memory 38 the instructions may specificallyconfigure the processing circuitry and, in turn, the computing device 34to perform one or more algorithms and operations described herein.

The memory 38 may include, for example, volatile and/or non-volatilememory. The memory 38 may comprise, for example, a hard disk, randomaccess memory, cache memory, flash memory, an optical disc (e.g., acompact disc read only memory (CD-ROM), digital versatile disc read onlymemory (DVD-ROM), or the like), circuitry configured to storeinformation, or some combination thereof. In this regard, the memory 38may comprise any non-transitory computer readable storage medium. Thememory 38 may be configured to store information, data, applications,instructions, or the like for enabling the computing device 34 to carryout various functions in accordance with example embodiments of thepresent disclosure. For example, the memory 38 may be configured tostore program instructions for execution by the processing circuitry 36.

The user interface 40 may be in communication with the processingcircuitry 36 and the memory 38 to receive user input and/or to providean audible, visual, mechanical, or other output to a user. As such, theuser interface 40 may include, for example, a display for providing animage captured by the camera 32 and/or an image visually depicting theclosest match between the candidate contacts and a predeterminedtemplate as described below. Other examples of the user interface 40include a keyboard, a mouse, a joystick, a microphone and/or otherinput/output mechanisms.

The communication interface 42 may be in communication with theprocessing circuitry 36 and the memory 38 and may be configured toreceive and/or transmit data, such as by receiving images from thecamera 32 and transmitting information, such as a list of candidatecontact insertion holes, contact ID numbers and locations of thecandidate contact insertion holes in a connector-based coordinatesystem, to a robot 44 and/or a robotic end-effector. Although referencedherein as candidate contact insertion holes, contact ID numbers andlocations of the candidate contact insertion holes, the list ofcandidate contact insertion holes, contact ID numbers and locations ofthe candidate contact insertion holes is to be interpreted so as to beassociated with the candidate contact insertion holes themselves and/orwire contacts aligned with the respective candidate contact insertionholes in those embodiments that include such wire contacts. Thecommunication interface 42 may include, for example, one or moreantennas and supporting hardware and/or software for enablingcommunications with a wireless communication network. Additionally oralternatively, the communication interface 42 may include the circuitryfor interacting with the antenna(s) to cause transmission of signals viathe antenna(s) or to handle receipt of signals received via theantenna(s). In some environments, the communication interface 42 mayalternatively or also support wired communication.

Referring now to FIG. 4, an example embodiment of a system performingthe methods described herein is shown including a robot end-effector 100including a first camera 102 and a second camera 104. The roboticend-effector 100 may carry a wire 110 including a wire contact at theleading end of the wire 110 in a wire gripper 108, while a connector 112is disposed in a fixed location as it is approached by the roboticend-effector 100. The two cameras 102, 104 are mounted on the roboticend-effector 100 in such a way as to view both the wire 110 includingthe wire contact and the connector 112 simultaneously.

While the embodiment of FIG. 4 includes two cameras, embodiments mayinclude more cameras. Further, a single camera may be used inconjunction with mirrors to observe different perspectives of the wirecontact and the connector using the single camera. Capturing multipleperspectives, such as using two or more cameras, may enable accuratepositioning of the wire contact and the connector as they are joined.

According to example embodiments described herein, images are capturedof the wire and wire contact along with the connector from more than oneperspective. Using the different perspectives, a line is identified thatextends in the direction of the wire and wire contact and a hole in theconnector that is the target hole for the wire is identified. FIG. 5illustrates images captured by cameras 102 and 104 of FIG. 4 of the wire110 including wire contact and the connector, specifically theidentified hole 124 of the connector into which the wire 110 is to beinserted. The line, identified through multiple perspectives, providesat least a stereoscopic indication of the relationship between the wirecontact and the target hole of the connector into which the wire is tobe inserted. Based on the identified line from the images, a movementcommand may be computed that would place the hole on the line in atleast two images. This may initially establish a rotation of theend-effector to bring the tip of the gripper 108 perpendicular to theconnector surface. To place the hole on the line, a movement isestablished in parallel to the connector surface to align the line withthe appropriate target hole of the connector. A movement command is thedesired displacement of the robot end-effector in three-dimensionalcartesian space. Aligning the wire contact with the hole places the wirein a proper position to enable the robot to move the wire along the linetoward the appropriate hole of the connector for insertion.

Embodiments described herein may calibrate the cameras ahead of usingthem to align the wire with the target hole of the connector. Thepurpose of the calibration is to compute a mapping of three dimensionalCartesian coordinates onto a two dimensional image coordinates. Thecalibration may be carried out before wires are fed to the roboticgripper of the end-effector. Calibration is not necessary before everywire insertion or before every connector change, but may be necessarywhen camera settings change, such as the focus, zoom, orientation, etc.

FIG. 6 depicts a process flow of an example calibration procedure. Theillustrated procedure uses a small calibration rod, which may be, forexample, a small plastic rod of around an inch in length and having adistinct tip such as a red tip. According to some embodiments, thecalibration rod may include a small sphere on a needle such as acomputer-aided measurement machine calibration stylus or simply a smalldot on a piece of paper. The calibration rod is used to provide aneasily identifiable point visible in each camera field of view from thedifferent camera perspectives. The calibration rod may be mounted firmlywith the tip facing up and within reach of the robot end-effector. Inpreparation for calibration, the robotic end-effector is advanced to bein front of the calibration rod, such that the calibration rod's tipposition mimics the expected position of a connector surface (e.g., thesurface into which the connector holes are formed). The calibrationprocedure may begin with a list of end-effector positions, as shown at130. One example for such a list are the three-dimensional coordinatesof nodes in a 3×3×3 cubic grid, where neighboring nodes are onecentimeter apart. A constraint for generating this list may be that foreach coordinate of the end-effector, the tip of the calibration rod mustbe visible in all camera images.

A new position of the end-effector positions is obtained at 132. Therobot loops through the list of end effector locations by moving therobot end-effector to the obtained position at 134, capturing images ofthe calibration rod at 136, finding the coordinates of the tip of thecalibration rod in both camera images at 138, and recording the endeffector position at 140. The process loops back to get a new positionfrom the list until all end-effector positions have been used forcalibration, or at least a predefined number of end-effector positionsto provide a satisfactory calibration. In each image captured, thelocation of the tip of the calibration rod is identified. To identifythe tip, the image may be color filtered (e.g., by computing R−(G+B)/2for each pixel, where R, G, and B are the Red, Green, and Blue colorchannels, respectively). The average location of all pixels having anintensity value above a predefined value may then be computed. Thepredefined value may be chosen such that only the tip of the calibrationrod is selected. It may be beneficial to have a light source above thecalibration rod such that the tip is sufficiently illuminated and maystand out in the captured images. The result of this calibrationprocedure are the two-dimensional image coordinates of the tip of thecalibration rod in each camera image.

Once the calibration routine of FIG. 6 is complete, the result is a listof three-dimensional end-effector positions in the end-effectorcoordinate frame and corresponding two-dimensional image coordinates foreach camera. This set of corresponding coordinates is used by analgorithm to calibrate each camera. For example, a Perspective-n-Point(PnP) algorithm may be used to calibrate each camera. A non-limitingexample of a PnP algorithm may be the UPnP+Gauss Newton algorithm. Theresult of this algorithm are two matrices for each camera: one thatencodes the intrinsic parameters (like the focal length) and one thatencodes the extrinsic parameters (position and pose of the camera).These matrices can be used to map a three-dimensional position in therobot's end-effector frame onto two-dimensional image coordinates. Usingthis calibration procedure, the camera locations do not need to be knownin advance.

Once the cameras have been calibrated, a wire contact held by therobotic end-effector may be aligned with a connector. FIG. 7 illustratesthe process of aligning a wire contact with a target insertion hole of aconnector. After a wire is grasped by the robotic gripper of theend-effector, whether the wire is placed in the gripper or picked-up bythe gripper, images may be captured by the cameras mounted on theend-effector at 150. In these images, the wire contact is detected andits direction obtained as shown in 152. This operation is furtherdescribed below. The robot may then move the wire contact to be near theconnector surface at 154. In this position, the cameras again captureimages at 156 to include the wire contact and the connector. From theseimages, two processes are computed: first the direction of the wirecontact is updated at 158; and second, connector holes are detected at160. By combining the output of these processes, the system computes amovement command in the robot end-effector coordinates to align thecontact with a target hole at 162.

After the robot executes the first alignment step, camera images areagain captured and both contact and target hole positions updated. Ifthis update yields a corrective movement command below a threshold(e.g., below 0.1 millimeters), the robot may not execute the correctionand instead proceeds to move the contact toward the connector surface.The direction of the movement of the wire contact toward the connectorsurface matches the contact's direction in three-dimensions as obtainedthrough the camera images. If the updated wire contact position yields acorrection above the threshold, the robot may then make the correctivemove and capture new images, whereby the aforementioned process isrepeated until the correction is below the threshold.

The number of repetitions of the process of FIG. 7 may be limited, suchas to three attempts. After this limit, the robot may abort thealignment process and indicate an error, such as through an errormessage of a user interface. Alternatively, the robot may start againmoving the contact near the connector surface as before. Threesignificant elements of the alignment process of FIG. 7 are described ingreater detail below.

Wire Contact Detection

The detection of the wire contact is necessary to align the contact witha target hole and to understand the movement direction for the robotend-effector once the contact is aligned. FIG. 8 depicts the process toextract the direction of a wire contact from an image. In this exampleembodiment, the computing device 34, such as the processing circuitry36, may be configured to perform the various operations of extractingthe direction of a wire contact and tip position from the capturedimages. The first operation is to extract a window of the image 166 inwhich the contact is expected to be. The image in the window may becolor filtered (e.g., by using a single color channel) to produce animage of only the color of interest. According to an example embodimentin which the wire contacts are gold in color, the image may be colorfiltered to find the gold colored areas at 167. A fit line isestablished at 168 based on the gold colored areas extending along alinear direction. The fit line constrains the area processed for edgedetection at 170, e.g., by using a 30-pixel wide corridor around theline. This corridor cuts out distracting edges in the background, e.g.,from other wires. For edge detection, the Canny edge detection algorithmmay be used. Non-limiting parameters of the Canny edge detector may be asigma or two for Gaussian blurring and thresholds of 0.005 and 0.015 foredge tracing. Second, to detect lines, a Hough transform may be carriedout on the edges as shown at 172. Third, using the resulting array fromthe Hough transform, the maximum may be found at 174 which correspondsto the longest line. By finding the maximum, the angle and orientationof the line and its distance from one of the image corners isidentified. Around the maximum, nearby maxima are sought with the sameline orientation. An example for these maxima is to have a value largerthan 0.5 times the maximum from the Hough transform. These maxima maycorrespond to parallel lines in the direction of the contact. The centerof the two extremal lines may be estimated as the position andorientation of the contact as shown at 176.

Once the direction of the contact is obtained, such as by the processingcircuitry 36 of the computing device 34, the location of the tip of thecontact is computed. To find the tip, the ends of all edge linesparallel to the contact may be determined. All ends may be projectedonto the contact line. The projection that is furthest away from theimage corner opposite the contact tip may be identified as the locationof the tip. This process to obtain the contact direction and tiplocation may be repeated for at least two camera images captured fromdifferent perspectives.

In the same way as for the wire contact, though without using the goldcolor filter described above, the tip of the robot gripper can beobtained by the processing circuitry 36 of the computing device 34.Here, images may be analyzed without the wire contact inserted in thegripper. Such images may be captured during calibration. Since thegripper is fixed relative to the cameras, the gripper-tip location canbe obtained as part of the calibration. Optionally, the gripper tiplocation can be computed in a process before a wire is gripped by thegripper.

Once the image coordinates of the gripper and contact tips are known inat least two camera views, the three-dimensional coordinates of the tipsmay be computed. To compute the three-dimensional coordinate of a point,virtual lines may be formed that extend from a camera location throughthe point in the image plane. These lines may be computed based on theextrinsic parameters of the cameras, as obtained during calibration. Thethree-dimensional coordinate may be obtained as the least-squaresolution that is closest to the virtual lines for at least two cameraviews. The direction of a contact in the three-dimensional end-effectorcoordinate frame may be computed as the vector difference between thecontact tip three-dimensional location and the gripper tipthree-dimensional location.

Contact Hole Detection

The contact hole detection is imperative to properly identify thecorrect hole of the connector into which the wire contact is to beinserted. In each camera image including the connector, contact holesare detected. FIG. 9 illustrates the corresponding process flow whichmay be performed by the processing circuitry 36 of computing device 34.At the beginning of the process, the robot end-effector is positioned infront of a connector such that the connector surface is fully visible inat least two camera images from different perspectives. For at least twoof the camera images from different perspectives, the below-describedprocess is followed.

An image is captured by a camera mounted to the robot end-effector at178. The image may be color filtered, such as using a red color filterby computing the R−(G+B)/2 for each pixel, where R, G, and B are thevalues for the red, green, and blue light channels respectively. Thecolor filter may be selected according to the color of a connector suchthat the filter best identifies differences in the connector surfacethat may correlate to holes of the connector. To crop the image, themedian image coordinate of the color-filtered intensity image iscomputed, and a window is cut out entered on the location of the median,as shown at 180. Alternatively, the window may be centered at the tip ofthe wire contact. The size of the window depends upon the type ofconnector and may be a pre-specified parameter, such as a size of270×270 pixels, sufficient to clearly identify each hole of theconnector. The color of the filter used should correspond to the colorof the connector.

The processing circuitry 36 may then be used to compute the squaredistance between a hole template and a local image patch from theintensity image for each patch location over the image as shown at 182.An example of a hole template may include an 18×18 pixel wide intensitygradient that mimics the shading inside a hole, where the intensityalong the gradient may follow a function f(x)=1/(1+exp(−x/1.7)). Theintensity of the template may be scaled to match minimum and maximumvalues of the color-filtered intensity image. This scaling increases therobustness to changes in lighting. The result of this operation mayinclude an intensity image in which low intensity areas (black) areareas of short distance to the hole template.

Using the intensity image may be used, such as by the processingcircuitry 36 of computing device 34, to isolate extrema identified inthe image. These extrema correspond to contact holes and are localminima in the distance image. To identify a local minimum, an ellipticalboundary around each pixel may be analyzed. The size of the boundary maydepend on the distance between neighboring holes on a connector. As anon-limiting example, the elliptical boundary may include half-axislengths of 25 and 15 pixels, where the longer axis is in the horizontaldirection approximating the elliptical shape of the holes in the cameraimage as the image is not coaxial with the connector. Pixels may bediscarded as extrema candidates if they have one or more pixels in theirboundary with an intensity below the candidate pixel's intensity times aconstant factor greater than one. This constant factor may depend on theconnector type. For example, the constant factor may be around 1.7. Afactor larger than one ensures that the extrema are more pronounced andmay eliminate or reduce false detections. For each remaining candidatepixel, a weighted average may be computed over all pixels inside itselliptical boundary, where the weight is the inverse of the intensity ofeach pixel in the distance image.

If a total weight computed over all pixels inside an elliptical boundaryis above a threshold (e.g., 2), then the weighted average may beidentified as a hole and added to the list. The detection of isolatedpeaks is shown at 184 whereby holes of the connector are identified. Toavoid that pixels of the same hole are counted as separate holes (doublecounting), all pixels inside an ellipse used for weighted averaging maybe marked and automatically discarded as extrema candidates. The resultof this operation is a list of contact holes. As an additionaloperation, outliers may be removed from the hole list. To removeoutliers, first the minimum distance (d_(min)) may be computed betweentwo holes. Second, any hole may be discarded as an outlier that has adistance to its nearest neighbor hole that is larger than a constantfactor times d_(min) (e.g., the constant factor of 2).

The hole list may be matched against known hole locations from technicalspecifications and/or drawings of the connector. This matching cancompensate for missed holes and allow for assignment of holeidentification numbers to the detected holes. From the technical drawingof a connector, a two-dimensional mask of hole locations may beextracted. This mask may include a list of contacts with the identitiesand locations in a connector-centered coordinate system. To match themask to the hole list, the mask may be rotated (in three axes) andtranslated (in three directions) in the end-effector coordinate systemsuch that it optimally overlaps with the hole list. To compute theoverlap, the mask may be projected onto each camera image using theparameters from the camera calibration. The cost function foroptimization may be the sum of square distance between the holes fromthe list and their closest neighboring projected holes. A non-limitingexample of an algorithm to optimize this cost function may includePowell's method.

Contact Alignment

The aforementioned processes provide, for each camera image analyzed,the line describing the wire contact and the location of the targethole. Based on this information, the corrective movement for the robotend-effector can be computed. The target hole location in two or morecamera images is identified at 200. The three-dimensional location “p”of the target hole in the end-effector coordinate system is computed at202. To compute this location, an optimization algorithm is used thatminimizes the sum of square distances between the target holetwo-dimensional image locations and the projections of thethree-dimensional location on to the camera images. A non-limitingexample for an optimization includes Powell's method. Here, thethree-dimensional location may be constrained to lie in the plane of theconnector surface. This plane may be known due to the mask-optimizationprocess described above, which rotates and translates the mask to matchthe connector surface.

A location “r” is computed in the end-effector coordinate system thatprojects closest to the contact line in each image as shown at 204. Thislocation may be also constrained to lie in the plane of the connectorsurface. An optimization algorithm may be used to compute “r”. Based onthe resulting values of “p” and “r”, the corrective movement may becomputed at 208 as c=p−r. The movement of the end-effector may becarried out at 210.

The identified target hole of the connector for the wire contact maythen be utilized to facilitate insertion of wire ends into respectivewire contact insertion holes of the connector. In this regard, a wiremay be identified by a wiring diagram or the like to be inserted into aparticular wire contact insertion hole of the connector (and, in someembodiments, also into electrical contact with a respective wire contactthat is aligned with the wire contact insertion hole) with theparticular wire contact insertion hole being identified by a contact IDnumber, which may be identified on the connector via the aforementionedmap of identifiers for the connector. Prior to insertion into the wirecontact insertion hole of the connector, a wire contact is generallyconnected to, e.g., crimped upon, a bare end of the wire to form a wireend. Based upon the contact ID numbers and corresponding locations ofthe candidate contact insertion hole for the connector 10, a wire endmay be inserted into the connector at the location associated with acontact insertion hole having the contact ID number of the wire contactinsertion hole 18 into which the wire is to be inserted. The computingdevice 34, such as the processing circuitry 36, may be configured todetermine the candidate contact insertion hole into which a wire is tobe inserted based upon the contact ID number of a candidate contactinsertion hole, such as based upon correspondence between the contact IDnumber of a candidate contact insertion hole and the contact ID numberof the wire contact insertion hole 18 into which the wire end is to beinserted as defined by a wiring diagram or the like. The computingdevice 34, such as the processing circuitry 36, is also configured todetermine the position of a robot 44 and, more particularly, a roboticend-effector utilized to insert the wire end into the candidate contactinsertion hole based upon the location of the candidate contactinsertion hole in the connector-based coordinate system and using thealignment methods described above for efficient and repeatable insertionof wires into corresponding holes of the connector.

As such, the computing device 34, such as the processing circuitry 36,may effectively drive a robot 44, such as a robotic end-effector, orotherwise provide information, such as a list of candidate contactinsertion holes, contact ID numbers and corresponding locations in theconnector-based coordinate system, to the robot sufficient to drive therobotic end-effector in such a manner as to insert the wire ends of awire bundle assembly into corresponding wire contact insertion holes 18.See, for example, FIG. 11 in which a plurality of wires 90 have beeninserted into respective wire contact insertion holes 18 of theconnector 10 in order to establish mechanical connection between thewire ends and the connector 10. By facilitating the automation of theconnection process associated with a wire bundle assembly, the system30, method and computer program product of an example embodimentincrease the efficiency with which wire ends of a wire bundle assemblymay be mechanically connected to a connector 10 and correspondinglyreduce the error rate and cost of the resulting assembly.

While the aforementioned process involves aligning a single wire contactfor insertion, embodiments of the present disclosure may be used toalign and insert a plurality of wire contacts into a respectiveplurality of target holes of a connector. However, the order in whichwire contacts are inserted into target holes may be established in sucha manner as to not diminish the effectiveness of the alignment methodsdescribed herein. As discussed above, the wire contact and the targethole must each be visible in at least two camera images from twodifferent perspectives for proper alignment. If wire contacts areinserted into the connector in an improper order, a target hole of theconnector may be obstructed from view of one or more cameras. As such,an order of assembly may include starting with target holes of theconnector which are furthest from the cameras, such as the bottom of theconnector in the example configuration shown herein. In this manner,wires will be inserted to the connector from the bottom-up to avoid aninserted wire obstructing the camera view of a target hole. A pluralityof cameras from a plurality of different perspectives may mitigate theinstallation order requirement as when a camera view of a target hole isobstructed, provided the target hole remains visible in at least twoimages from at least two perspectives, the process described herein canbe performed effectively.

FIG. 12 is a flowchart of a process for aligning a wire contact with atarget hole of a connector according to an example embodiment of thepresent disclosure. As shown, images are obtained from at least twoimage capture devices, such as cameras 32 of apparatus 30, attached toan end-effector of a robot, where the images are of a wire gripper ofthe end effector, at 220. At 222, a wire contact is detected, such as byprocessing circuitry 36 of computing device 34, within at least oneimage from each of the at least two image capture devices. Within atleast one image from each of the at least two image capture devices, oneor more insertion holes of the connector are detected at 224, such as byprocessing circuitry 36 of computing device 34. Corrective movement ofthe robot end effector is identified at 226, such as by processingcircuitry 36 of computing device 34, that aligns a target hole of theone or more insertion holes of the connector with the wire contact. At228, the robot (44 of apparatus 30 of FIG. 3) is caused, such as byprocessing circuitry 36 via communications interface 42, to move theend-effector according to the identified corrective movement.

Once the alignment of the wire contact with the target hole of theconnector is performed, the wire contact may be inserted into the targethole for assembly of the connector and wire bundle thereof. Initially, awire contact may be moved by the gripper of the end-effector of therobot to a preparation position proximate the target hole. The alignmentis performed in a plane parallel to the surface of the connector, whileinsertion is performed on an axis orthogonal to the plane of the surfaceof the connector. The wire contact may be moved a predetermined distancealong the insertion axis to the target hole of the connector whereby ameasured force on the gripper is used to establish proper insertion ofthe wire contact. The wire contact may be moved the predetermineddistance from the preparation position, where the predetermined distanceis a distance that would result in contact with the connector in theevent of misalignment, or partial insertion into the target hole in theevent of proper alignment. Optionally, the robot end-effector may beadvanced along the contact line until exceeding a force thresholdestablishing contact with a surface, either being mis-aligned and havingmissed a hole, or being partially inserted into a target hole. Thegripper may measure the force on the gripper upon moving thepredetermined distance to establish if the force is above a surfacecontact threshold, which may be around 0.3 to 0.5 Newtons. The processmay then test to establish if the contact is inserted in the target holeby measuring forces while moving the initial predetermined distance. Theprocess may abort if a force greater than the threshold is measured,which may be indicative of misalignment between the wire contact and thetarget hole. If it is determined that the wire contact was not properlyaligned with the target hole, the wire contact may be moved axially awayfrom the connector by the robot and the end-effector. The alignmentprocess described above may be performed again to establish properalignment and corrective transformation of the wire contact in the planeparallel to the surface of the connector.

The initial distance for the preparation position and the thresholdforce described above may be empirically determined and established suchthat the wire contact would not penetrate the rubber grommet of theconnector in the event of misalignment. If the initial move from thepreparation position along the insertion axis the predetermined distanceis completed without exceeding the force threshold, then alignment iscorrect and the wire contact has successfully entered the target hole.If the robot advances a predefined distance, such as three millimeters,beyond the point where the wire contact would first touch the connector,if during this movement a force threshold (e.g., four Newtons) has notbeen exceeded, then it may be determined that the wire contact hassuccessfully entered the target hole. An estimate for the target hole'sposition may then be calculated based on the contact's orientationvector, the initial insertion depth commanded, and an additional offsetempirically determined per contact-connector pair (e.g., 0 to 8millimeters). This additional offset may be necessary due to surfacecontact being detected past the connector's actual surface plane independence on the hole, contact geometry, and friction.

In-Hole Contact Alignment and Insertion

Once the tip of the contact has been inserted into the desired hole viathe operations described above, an initial alignment correction may beperformed by aligning the wire contact's orientation vector with theconnector surface normal. This alignment correction may be computed byprojecting the contact orientation vector onto a plane parallel to theconnector surface. The purpose of this alignment correction is tocorrect for contacts that are bent to one side at the tip of thegripper. After alignment correction, the distance from the gripper fromthe connector's surface is calculated using the hole's positionestimate. This distance may be used to determine the allowable distancethe gripper can move without penetrating the connector past theconnector's limit. Once the allowable distance is determined, thegripper may be commanded to insert the contact into the target hole bytraveling the distance along the connector's surface normal/gripperorientation vector, or until an insertion force threshold is reached,such as twelve Newtons.

If the movement is executed without exceeding the force threshold, thegripper is as close to the connector as is allowable, and thus analternative means of pushing the contact and wire forward may be neededto achieve full insertion. This may be performed through a “re-grip”operation. If the movement of the gripper is stopped due to the forcethreshold, then the distance that the gripper has traveled is used todetermine if the wire could have been inserted. If the distance that thegripper has traveled is below a specified parameterized distance, thenit may be established that the wire contact is not properly and fullyinserted and seated within the connector, such that the force thresholdwas reached for a different reason, such as an obstruction within theconnector. If the calculated distance indicates that the wire was likelyinserted and seated by being past the specified parameterized distance,then a pull test may be performed to determine success or failure of theinsertion process.

If the distance moved by the gripper holding the wire contact hasindicated that insertion is not yet complete, then the number ofinsertions tried may be checked to identify any issues. If the number ofinsertions attempted exceeds a parameterized value, the insertionprocedure may end and the insertion procedure may be considered afailure. If the number of insertion attempts is not above theparameterized value, a visual alignment operation may be performed toalign the in-hole contact and the insertion process may be repeated. Thevisual alignment method used in an example embodiment may be the samemethod used to estimate the contact vector. The difference may includethat the image analysis is constrained to a small image window (e.g.,100×100 pixels) near the tip of the robotic gripper, and that the lengthof the visible portion of the contact may be determined by intersectingthe contact vector with the connector surface. The resulting vector maybe used to compute a corrective movement in a coordinate frame of therobotic end-effector. The corrective movement may be computed such thatit aligns the contact vector with the surface normal of the connectorsurface. This surface may be known due to the mask-optimization processdescribed above.

When the gripper has advanced a wire contact as far as the gripper ispermitted to move (e.g., when the gripper is as close to the connectoras allowed), and the wire contact has not yet reached full insertion andseating, the gripper may perform a re-grip operation. For the re-gripand insertion operation, the gripper reduces the friction forces used tohold the wire in the gripper and moves back by a parameterized distance(e.g., 4 mm). The gripper then re-grips the wire and pushes the wirefurther into the connector by returning to its original position. Thegripper will stop moving if it reaches its closest allowable proximityto the connector or if the insertion force threshold is reached.

Once a wire is fully inserted and seated, a pull test operation may beperformed to confirm seating of the wire contact within the connector.For the pull test, the gripper may pull back on the wire, away from theconnector, until a specific distance or force threshold is reached. Ifthe force threshold is reached before the specified distance, then thewire contact is confirmed as properly seated. If the specified distanceis reached before the force threshold is achieved, the wire insertionfailed as the wire is determined to not be fully seated.

FIG. 13 illustrates a process flow according to the aforementioneddescribed method of alignment and insertion of a wire contact with atarget hole of a connector. As shown, the process begins at 300 with aconnector mounted and ready to receive a wire contact in a target hole.The wire including the wire contact is loaded at 302 into a wire gripper(e.g., 108 of FIG. 4) of an end-effector (e.g. 100) of a robot. Initialvision processing is performed at 304, such as using cameras 102 and 104of FIG. 4. The end-effector may then move the wire gripper and wire tothe preparation position 306. At 308, alignment of the wire contact withthe target hole is performed. As shown, the corrective transformation iscomputed at 310, using the process described in detail above. If thecorrection transformation is below a threshold amount, the alignment isconsidered complete and the process continues. If the correctiontransformation is above a threshold, the number of correctivetransformations already completed is checked at 312, and if it is belowa threshold number, the corrective transformation occurs at 314. If thenumber of corrective transformations exceeds a predefined number, afailure may be identified at 330. However, if the correctivetransformation is successful at 314 and results in an alignment below athreshold corrective transformation, then the wire gripper holding thewire contact is moved toward the connector at 316.

The process of FIG. 13 continues at 318 whereby movement of the wiregripper toward the connector to a position in which misalignment wouldcause contact but not damage to the wire contact or connector. If forcefeedback on the wire gripper exceeds a threshold value, it may bedetermined that the wire contact is not aligned with the target hole,such that the wire contact is retracted at 320 and alignment is againperformed at 308. If the force feedback at the wire gripper is below athreshold, the wire contact is established as being aligned with thetarget hole at 322. The wire gripper may then advance the wire contactto a maximum distance allowed by the wire gripper at 324. The maximumdistance may be a distance at which the wire gripper does not contactthe connector, or contacts the connector but does not damage theconnector. In this way, the wire gripper is prevented from damaging theconnector during insertion of the wire contact.

If the maximum distance of the wire gripper is reached before aninsertion force threshold is observed at the wire gripper, the wiregripper may release the wire contact, while the end-effector isretracted with the wire gripper to re-grip the wire and further advancethe wire contact into the target hole of the connector at 322. Once theinsertion force threshold is reached, it is confirmed at 326 whether ornot the wire contact was advanced at least a minimum distance into theconnector. If the distance the wire contact was advanced was below athreshold distance, an obstruction within the connector may haveoccurred and alignment may be again attempted at 322. If the insertiondistance was satisfied at 326, a pull test may be conducted. The pulltest uses the wire gripper to pull the wire away from the connector. Ifthe force at the wire gripper satisfies a pull test threshold withoutthe wire contact moving from the connector (e.g., above a smallthreshold distance), then the wire contact is established as fullyinserted at 334, and success is indicated at 336.

FIG. 14 is a flowchart of a method for aligning and inserting a wirecontact into a target hole of a connector. The illustrated embodimentincludes obtaining captured images from at least two image capturedevices attached to an end-effector, the images of a wire gripper of theend-effector at 400. The captured images are processed at 402 toestablish a corrective transformation to align the wire contact with thetarget hole of the connector. The robot translates the end-effector at404 to move the wire griper along the corrective transformation. Therobot is caused to advance the end-effector until contact is made withthe connector surface at 406. Based on the force measurements at the endeffector, as shown at 407, it is identified if the wire contact isinside the target hole of the connector. At 408, the robot is caused tomove the end-effector toward the connector a predetermined additionalamount more than the predetermined distance. Based on the movement ofthe wire contact the predetermined additional amount more than thepredetermined distance, at 410 it is identified if alignment is correctfrom force feedback at the wire gripper.

As described above, FIGS. 6, 7, 10, 12, and 14 illustrate flowcharts ofa system 30, method, and computer program product according to exampleembodiments of the present disclosure. It will be understood that eachblock of the flowcharts, and combinations of blocks in the flowcharts,may be implemented by various means, such as hardware, firmware,processor, circuitry, and/or other devices associated with execution ofsoftware including one or more computer program instructions. Forexample, one or more of the procedures described above may be embodiedby computer program instructions. In this regard, the computer programinstructions which embody the procedures described above may be storedby the memory 38 of a system 30 employing an embodiment of the presentdisclosure and executed by the processing circuitry 36 of the system 30.As will be appreciated, any such computer program instructions may beloaded onto a computer or other programmable apparatus (e.g., hardware)to produce a machine, such that the resulting computer or otherprogrammable apparatus implements the functions specified in theflowchart blocks. These computer program instructions may also be storedin a computer-readable memory that may direct a computer or otherprogrammable apparatus to function in a particular manner, such that theinstructions stored in the computer-readable memory produce an articleof manufacture the execution of which implements the function specifiedin the flowchart blocks. The computer program instructions may also beloaded onto a computer or other programmable apparatus to cause a seriesof operations to be performed on the computer or other programmableapparatus to produce a computer-implemented process such that theinstructions which execute on the computer or other programmableapparatus provide operations for implementing the functions specified inthe flowchart blocks.

Accordingly, blocks of the flowcharts support combinations of means forperforming the specified functions and combinations of operations forperforming the specified functions for performing the specifiedfunctions. It will also be understood that one or more blocks of theflowcharts, and combinations of blocks in the flowcharts, can beimplemented by special purpose hardware-based computer systems whichperform the specified functions, or combinations of special purposehardware and computer instructions.

In some embodiments, certain ones of the operations above may bemodified or further amplified. Furthermore, in some embodiments,additional optional operations may be included. Modifications,additions, or amplifications to the operations above may be performed inany order and in any combination.

Many modifications and other embodiments of the inventions set forthherein will come to mind to one skilled in the art having the benefit ofthe teachings presented in the foregoing descriptions and the associateddrawings. Therefore, it is to be understood that the present applicationis not to be limited to the specific embodiments disclosed and thatmodifications and other embodiments are intended to be included withinthe scope of the appended claims. Moreover, although the foregoingdescriptions and the associated drawings describe example embodiments inthe context of certain example combinations of elements and/orfunctions, it should be appreciated that different combinations ofelements and/or functions may be provided by alternative embodimentswithout departing from the scope of the appended claims. In this regard,for example, different combinations of elements and/or functions thanthose explicitly described above are also contemplated as may be setforth in some of the appended claims. Although specific terms areemployed herein, they are used in a generic and descriptive sense onlyand not for purposes of limitation.

That which is claimed is:
 1. A system to insert wire contact into atarget hole of a connector, the system comprising: a robot having anend-effector, wherein the end-effector comprises a wire gripper and atleast two image capture devices secured to the end-effector; a computingdevice, wherein the computing device is configured to: process imagescaptured by the image capture devices to establish a correctivetransformation to align the wire contact with the target hole of theconnector in a connector surface; cause the robot to translate theend-effector to move the wire gripper along the correctivetransformation; cause the robot to advance the end-effector untilcontact is made with the connector surface; identify, based on forcemeasurements, if the wire contact is inside the target hole of theconnector; cause the robot to advance the end-effector to move the wirecontact toward the connector a predetermined additional amount; andidentify, based on movement of the wire contact the predeterminedadditional amount, if alignment is correct from force feedback at thewire gripper.
 2. The system of claim 1, wherein the computing deviceconfigured to identify, based on force measurements, if the wire contactis inside the target hole of the connector, is configured to: identifythat the wire contact is not aligned with the target hole of theconnector in response to a force observed at the wire gripper above apredefined value; and identify that the wire contact is aligned with thetarget hole of the connector in response to the force observed at thewire gripper below the predefined value.
 3. The system of claim 1,wherein the computing device configured to process images captured bythe image capture devices to establish corrective transformation toalign the wire contact with the target hole of the connector isconfigured to: identify, within the processed images, a tip of the wirecontact and a direction along which the wire contact extends; andestablish a corrective transformation to align the tip of the wirecontact and the direction along which the wire contact extends with thetarget hole in an axis of a three-dimensional coordinate system of therobot end-effector from the processed images.
 4. The system of claim 1,wherein the computing device is further configured to: determine amaximum distance the wire gripper can advance toward the connector;cause the robot to advance the end-effector to move the wire grippertoward the connector; as the robot is caused to advance the end-effectorto move the wire gripper toward the connector, in response to forcefeedback on the wire gripper satisfying an insertion force value,determine that the wire contact is fully inserted into the connector;and as the robot is caused to advance the end-effector to move the wiregripper toward the connector, in response to force feedback on the wiregripper failing to satisfy the insertion force value before reaching themaximum distance the wire gripper can advance toward the connector,determine that the wire contact is not fully inserted into theconnector.
 5. The system of claim 4, wherein in response to thecomputing device determining that the wire contact is fully insertedinto the connector, the computing device is further configured to: causethe robot to retract the end-effector to move the wire gripper away fromthe connector; and in response to the wire gripper moving a pulldistance without force feedback on the wire gripper exceeding a pulltest value, identify the wire contact as improperly inserted; inresponse to force feedback on the wire gripper exceeding a pull testvalue without the wire gripper having moved a pull distance, identifythe wire contact as properly inserted.
 6. The system of claim 4, inresponse to the computing device determining that the wire contact isnot fully inserted into the connector, the computing device is furtherconfigured to: cause the wire gripper to release a grip on the wirecontact; cause the robot to retract the end-effector to move the wiregripper away from the connector; cause the wire gripper to re-grip thewire contact; and cause the robot to advance the end-effector to movethe wire gripper and the wire contact toward the connector.
 7. Thesystem of claim 1, wherein force feedback at the wire gripper isestablished based on a force sensor between the robot and the wiregripper.
 8. The system of claim 1, wherein the computing device isfurther configured to: identify, in images captured by the image capturedevices, an orientation of a visible portion of the wire contact; andcompute a corrective movement in a coordinate frame of the robotend-effector using the orientation of the wire contact.
 9. A method toalign and insert a wire contact into a target hole of a connector, themethod comprising: obtaining captured images, from at least two imagecapture devices attached to an end-effector of a robot, of a wiregripper of the end-effector; processing images captured by the imagecapture devices to establish a corrective transformation to align thewire contact with the target hole of the connector in a connectorsurface; causing the robot to translate the end-effector to move thewire gripper along the corrective transformation; causing the robot toadvance the end-effector until contact is made with the connectorsurface; identify, based on force measurements, if the wire contact isinside the target hole of the connector; causing the robot to advancethe end-effector to move the wire contact toward the connector apredetermined additional amount more; and identifying, based on movementof the wire contact the predetermined additional amount more, ifalignment is correct from force feedback at the wire gripper.
 10. Themethod of claim 9, wherein identifying, based on force measurements, ifthe wire contact is inside the target hole of the connector, comprises:identifying that the wire contact is not aligned with the target hole ofthe connector in response to a force observed at the wire gripper abovea predefined value; and identifying that the wire contact is alignedwith the target hole of the connector in response to the force observedat the wire gripper below the predefined value.
 11. The method of claim9, wherein processing images captured by the image capture devices toestablish corrective transformation to align the wire contact with thetarget hole of the connector comprises: identifying, within theprocessed images, a tip of the wire contact and a direction along whichthe wire contact extends; and establishing a corrective transformationto align the tip of the wire contact and the direction along which thewire contact extends with the target hole in an axis of athree-dimensional coordinate system of the robot end-effector from theprocessed images.
 12. The method of claim 9, further comprising:determining a maximum distance the wire gripper can advance toward theconnector; causing the robot to advance the end-effector to move thewire gripper toward the connector; as the robot is caused to advance theend-effector to move the wire gripper toward the connector, in responseto force feedback on the wire gripper satisfying an insertion forcevalue, determining that the wire contact is fully inserted into theconnector; and as the robot is caused to advance the end-effector tomove the wire gripper toward the connector, in response to forcefeedback on the wire gripper failing to satisfy the insertion forcevalue before reaching the maximum distance the wire gripper can advancetoward the connector, determining that the wire contact is not fullyinserted into the connector.
 13. The method of claim 12, wherein inresponse to determining that the wire contact is fully inserted into theconnector, the method further comprising: causing the robot to retractthe end-effector to move the wire gripper away from the connector; andin response to the wire gripper moving a pull distance without forcefeedback on the wire gripper exceeding a pull test value, identifyingthe wire contact as improperly inserted; in response to force feedbackon the wire gripper exceeding a pull test value without the wire gripperhaving moved a pull distance, identifying the wire contact as properlyinserted.
 14. The method of claim 12, in response to determining thatthe wire contact is not fully inserted into the connector, the methodfurther comprising: causing the wire gripper to release a grip on thewire contact; causing the robot to retract the end-effector to move thewire gripper away from the connector; causing the wire gripper tore-grip the wire contact; and causing the robot to advance theend-effector to move the wire gripper and the wire contact toward theconnector.
 15. The method of claim 9, wherein force feedback at the wiregripper is established based on a force sensor between the robot and thewire gripper.
 16. The method of claim 9, further comprising:identifying, in images captured by the image capture devices, anorientation of a visible portion of the wire contact; and computing acorrective movement in a coordinate frame of the robot end effectorusing the orientation of the wire contact.
 17. A computer programproduct for aligning and inserting a wire contact into a target holedefined by a connector, the computer program product comprising at leastone non-transitory computer-readable storage medium havingcomputer-executable program code instructions stored therein, thecomputer-executable program code instructions comprising program codeinstructions to: obtain captured images from at least two image capturedevices attached to an end-effector of a robot, of a wire gripper of theend-effector; process images captured by the image capture devices toestablish a corrective transformation to align the wire contact with thetarget hole of the connector in a connector surface; cause the robot totranslate the end-effector to move the wire gripper along the correctivetransformation; cause the robot to advance the end-effector untilcontact is made with the connector surface; identify, based on forcemeasurements, if the wire contact is inside the target hole of theconnector; cause the robot to advance the end-effector to move the wirecontact toward the connector a predetermined additional amount morethan; and identify, based on movement of the wire contact thepredetermined additional amount more, if alignment is correct from forcefeedback at the wire gripper.
 18. The computer program product of claim17, wherein the program code instructions to identify, based on forcemeasurements, if the wire contact is inside the target hole of theconnector, comprises program code instructions to: identify that thewire contact is not aligned with the target hole of the connector inresponse to a force observed at the wire gripper above a predefinedvalue; and identify that the wire contact is aligned with the targethole of the connector in response to the force observed at the wiregripper below the predefined value.
 19. The computer program product ofclaim 17, wherein the program code instructions to process imagescaptured by the image capture devices to establish correctivetransformation to align the wire contact with the target hole of theconnector comprises program code instructions to: identify, within theprocessed images, a tip of the wire contact and a direction along whichthe wire contact extends; and establish a corrective transformation toalign the tip of the wire contact and the direction along which the wirecontact extends with the target hole in an axis of a three-dimensionalcoordinate system of the robot end-effector from the processed images.20. The computer program product of claim 17, further comprising programcode instructions to: determine a maximum distance the wire gripper canadvance toward the connector; cause the robot to advance theend-effector to move the wire gripper toward the connector; as the robotis caused to advance the end-effector to move the wire gripper towardthe connector, in response to force feedback on the wire grippersatisfying an insertion force value, determine that the wire contact isfully inserted into the connector; and as the robot is caused to advancethe end-effector to move the wire gripper toward the connector, inresponse to force feedback on the wire gripper failing to satisfy theinsertion force value before reaching the maximum distance the wiregripper can advance toward the connector, determine that the wirecontact is not fully inserted into the connector.